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ABSTRACT 

As part of the National Study of American Indian 
Education, this report assessed the validity of the analysis of 
interview and guestionnai re data obtained. With some significant 
exceptions, agreement was good between the rating scale and 
questionnaire analysis and the field workers* observations on ranking 
and comparing the 4 schools; 3 public schools serving Indian 
communities on the Olympic Peninsula in Washington and 1 
state-operated school in Bethel, Alaska. (See other papers in Series 
IV of the National Study, especially numbers 1, 7, 9, 10, 11, and 

12.) Discrepancies revealed in comparison of results from the 2 types 
of evaluation were that (1) in evaluating parental opinions of the 
school administration in the schools, the field team was in marked 
disagreement with the scale results; (2) in ranking the 4 schools 
according to the teachers' "understanding of the local Indian 
community," there was no agreement on the ranking of 3 of the 4 
schools; (3) the scales sometimes failed to differentiate between 
schools where the field workers felt they could; and (4) the field 
workers sometimes felt that no meaningful dif fere ntiation between, or 
ranking of, schools could be done where it had been done with the 
scale results. A final conclusion was that field workers need to add 
their observations and interpretations to the results of the 
interview and questionnaire analysis in preparing the reports of this 
study. (LS) 




00*6" 



AMERICAN INDIAN EDUCATION 



N. 

oo 

r- 

-tr 

o 

Q 

UJ 



NATIONAL STUDY OF 



USOE 

PROJECT OEC- 



0-8-080147-2805 



FINAL REPORT 



Series IV 
Nob 8 



Perc eptions 



of indxan L l uccttion 



„ validity o£ Rating Scales and Xnterviens 
■for Evaluating Indian Education 




US DEPARTMENT OF HEALTH. 

^EDUCATION StWELFARE 

OFFICE OF E°UCATI0N 
THIS document has men o R6M om 
duced exactly as “WV 0R1G 

the person or organiza opin 

,NAT,N pTb T co° dSnotnecessar.lv 
' b °e N fresent E Sfrc.al office of edu- 

S POSITION OR POLICY. 



Bruce A. Birchard 
Research Assistant 
University of Chicago 

December, 1970 



cERiC 



1 



■NATIONAL STUDY OF AMERICAN INDIAN EDUCATION 



The attached paper is one of a number which make up the Final 
Report of the National Study of American Indian Education. 

This Study was conducted in 1968-69-70 with the aid of a grant 
from the United States Office of Education^ GEC-0-8-080I47-2805. 



The Final Report consists of five Series of Papers: 



I. Community Backgrounds of Education in the Communities 
Which Have Been Studied. 

II. The Education of Indians in Urban Centers. 

III. Assorted Papers on Indian Education--mainly technical 
papers of a research nature. 

IV. The Education of American Indians--Substantive Papers. 

V. A Survey of the Education of American Indians. 



The Final Report Series will be available from the ERIC Document 
Reproduction Service after they have been announced in R esearch in 
Education . They will become available commencing in August, 1970, and 
the Series will be completed by the end of 1970. 
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'The Validity of Rating Scales and Interviews 



for Evaluating Indian Education* 



The problem V7e set ourselves in preparing this report is to try to assess 
the validity of our analysis of the interview and questionnaire data obtained 
in the National Study of American Indian Education. We want to know how closely 
our analysis and interpretation of the variables we have attempted to measure 
corresponds to the "real, situation*" Like any social scientist, we have no 
perfect means of determining what "really is.' 1 In our case, the best we can 
do is to assess the "concurrent validity" by comparing two different analyses 
of the same set of several communities . From the earliest stages of planning 
in the National Study of American Indian Education, efforts have been made to 
insure that two basic techniques of data-gathering and analysis would be 
employed: (1) through instruments such as interviews, questionnaires, inven- 

tories and tests, and (2) through direct observation in the field. We can 
therefore carry out a comparison of the results of these two approaches in an 
effort to measure the concurrent validity. 

We could not possibly test the degree of concurrence between the interview 
and questionnaire analysis and the observations of our dozens of field workers 
in 7 different field centers. We had to content ourselves with comparing our 
analysis with the observations of the three principal field investigators from 
one center--the "San Francisco field team." The three principal investigators 
in this team were: Dr. John Connelly, director, and Dr. Ray and Mrs. Carol 

Barnhardt, assistants to Dr. Conr.ellv in the field. These individuals spent 
six to eight weeks in each of the five communities they studied. Connelly and 
Ray Barnhardt had had considerable training and experience in techniques of 
field observation. 
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*This report is one of a series on the results of the rating scale analysis 
done using Student, Parent, Community Leader and Teacher Interviews recorded by 
personnel on the National Study of American Indian Education. The reader will 
understand what is discussed in this paper belter if he is familiar with the 
other papers in Series IV of the Final Report of the National Study of American 
Indian Education, particularly numbers 1, 7, 8, 9, 10, 11 and 12. Paper No. 1 
lists the schools studied and some of their characteristics and the nature of 
the field research. Paper No. 7 includes examples of the interview schedules 
and rating scales utilized in the Study and describes how they were formulated 
and used. Papers No. 8 and No. 9 deal with the validity and reliability of the 
rating scales and interviews. Numbers 11 and 12 are based on the results of 
the rating scale and questionnaire analyses. Number 10 deals with the attitudes 
of Indians toward their culture. Number 11 reports on the evaluations Indians 
make of their schools. Number 12 discusses the attitudes of Indians toward 
formal education, of teachers toward Indians, and the communication and inter- 
action between Indians and teachers. 



3 



2 



We began the; test by analyzing student, parent, teacher, and community 
leader interviews (using our rating scales), several items from the semantic 
differential for students, and several items from the Teacher Questionnaire. 

(See Paper #7 in this series for descriptions and examples of these items). 

After completing this analysis, we drew up a list of questions designed to 
elicit answers from our field investigators which could be compared with the 
results of the rating scale and questionnaire analysis. This list of questions 
was sent to Dr. Connelly and to the Barnliardts without their having seen our 
results. These two parties replied separately to the questions and returned 
their responses to us. We then read and closely compared the results of the 
rating scale aud questionnaire analysis with the analysis they had made of 
their observations. We were particularly interested in how they ranked the 
four schools in question on a number of variables, since this was what v?e 
intended to do with the total sample of schools in Papers Wo. 10, No. 11, and 
No. 12 of this series. 

A word on the four schools is in order at this point. Three of them are 
public schools serving coastal reservation Indian communities on the Olympic 
Peninsula in Washington. At Neah Bay, a school with grades kindergarten to 
12 enrolls roughly 300 students, 70 percent of whom are Makah Indians. Quinault 
Indians at Taholah send their children to two schools. The Taholah school 
enrolls 130 Quinault children in pre-school classes through sixth grade. This 
school has been given considerable recognition as an excellent elementary school, 
having been rated one of the "ten best" in the United States a few years ago. 
Taholah children in grades seven to twelve are bussed off the reservation to 
.the neighboring town of Moclips, where they constitute 30 percent of the student 
body in those grades. The fourth school used for this report was the state- 
operated school in Bethel, Alaska, enrolling nearly 800 students, of whom 90 
percent are Eskimo. This is the least acculturated community of the three in 
our sample, but all are similar in being relatively isolated from the urban- 
suburban stream of modern American life. 

We shall now turn to the question at hand- -to what extent do our two 
analyses agree and to what extent do they disagree? We shall consider the 
results of our comparison under several headings which demarcate the general 
topics or issues which our data covered. These are the topics which are 
covered in the final three papers in this series (Nos. 10, 11, and 12). For 
these three papers, data from all seven field centers were analyzed using the 
rating scales and questionnaire items reported on here. 



Attitudes and Opinions Regarding Tribal Culture 

We concluded, on the basis of the rating scale and questionnaire results, 
that most students and most parents in all communities were interested in their 
local native culture, and that most liked the idea that this be included in 
some way in school. It appeared, however, that only one of the four schools 
was seen by its Indian community as doing anything to help Indian students 
learn about their native culture or history. We also concluded that few students 
or parents had given much thought to either the specific question of how children 
could be taught about their tribal heritage or the more general question of the 
nature of the assimilation process. The field team*s observations support these 
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generalizations. The field observers were also able to rank the three com- 
munities as to the degree of interest in the native culture and language 
relative to each other; this was not possible with the rating scale results* 

Using the results on the Teacher Questionnaire , we arrived at some con- 
clusions about the teachers. First of all, it appeared that more than half 
of the teachers disagreed with the statement that Indians should become com- 
pletely assimilated. Instead, a "pluralls tic emphasis 1 ’ was stressed by most. 
Nevertheless, most teachers quite clearly saw the most important function of 
the school- to be the preparation of its students for successful lives in the 
dominant society. Tile field team : s interpretations add to this analysis. The 
members of the field team all felt that, though many teachers gave lip service 
to cultural pluralism, few either understood or acted upon it. Almost all 
worked toward the goal of as s imi la t ion. This is a case in which the field 
workers and the rating scales were in basic agreement, but in which the field 
workers gave a more explicit interpretation of the situation. 

It is impossible for us to evaluate the field workers' specific feelings 
that few teachers understood or acted upon their verbal commitment to cultural 
pluralism. Clearly, they had their own idea of what cultural pluralism is-- 
one which they felt few teachers understood — and this clearly constitutes a 
bias in the sense that anyone who interprets or evaluates something must do so 
against his own standard. 

From the responses to three other questions, we gathered that teachers 
varied widely in their opinions as to the divergence between what the school 
taught Indian students and what they learned in their homes. Some indicated 
that they saw little conflict; others felt that considerable conflict existed 
and that Indian students should be ’’wooed away” from their parents' influence. 

The field workers agreed, but they also felt that many teachers saw the typical 
Indian child's home life as a severe handicap to good school performance. 

According to the field team, many teachers believed that Indian parents wanted 
their children to get a good education, but these same teachers still saw the 
home background of these students as a significant drawback to their success 
in school. 

Based on the responses to two questions on the teacher questionnaire, we 
concluded that Bethel and Taholah teachers were more concerned with the use 
of local cultural materials In the curriculum than Neah Bay and Moclips teachers. 
The field workers, on the other hand, felt that no significant differences between 
the teachers in the four schools could be claimed. It was also the field team's 
opinion that, though some teachers were concerned about the incorporation of 
native culture into the curriculum, none understood the issue very well and none 
would act to make any significant changes. This is again a judgment of the field 
workers which stems to some extent from their own ideas about cultural differences 
The findings from the interview and questionnaire analysis that most teachers 
are overwhelmingly in favor of seeing some course (s) In Indian history and culture 
started in school and are interested in incorporating aspects of the local culture 
into the standard curriculum arc not contradicted by the field team's observations 
They would simply point out that most teachers have little understanding of what 
the local Indian culture is. 
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We concluded, after interpreting the scale and questionnaire results, 
that teachers and community leaders appear to have given this matter of the 
incorporation of tribal history and culture into the school curriculum more 
thought than most students and parents, and they might well be the stronger 
force for such a change at this point. The field workers, on the other hand, 
were less willing to generalize. They felt that the impetus for programs 
dealing with tribal culture and history came from different groups (students, 
parents, teachers, and community leaders) in different communities . Unfortu- 
nately, they did not agree very well on which groups were most concerned about 
this in each community. They clearly did not feel that one could conclude 
that teachers and influential people were generally more concerned than most 
students and parents, however. 



Approval and Disapproval of the School 

Students: There was basic agreement between the overall scale and semantic 

differential results and the evaluation of the members of the field team that 
Mo clips students were on the whole, less happy with their school than were 
students in the other three schools. On the basis of the scale results, we 
also gathered that Bethel students had fewer complaints about their school 
than students in other schools. The Earnhardts arrived at the same conclusion, 
though Connelly did not differentiate students at Taholah, Neah Bay, and Bethel 
on this dimension. 

Parents: Again there was general agreement between the scales and the 

field team that parents were least pleased with the Moclips school. The 
Barnhardts also agreed with the -scale results in stating that Taholah parents 
approved most highly of their school (Taholah Elementary), though Connelly did 
not differentiate Taholah parents from Neah Bay and Bethel parents on this 
dimension. 

Connelly agreed with the scale results that little serious criticism of 
the curriculum existed. He did sense that many teachers, students, and parents 
were dissatisfied with the curriculum in a general way, but they heard few 
specific or meaningful criticisms or suggestions for improvement. The Barnhardts, 
on the ether hand, felt that "most parents agreed that the curriculum was in- 
appropriate for Indian students. 11 They may be referring primarily to a feeling 
among parents that the curriculum should include some work on Indian history 
and culture; this much was evident from the rating scale analysis of the * iter- 
views as well. 

On the scale dealing with parent approval or disapproval of teachers, it 
appeared that Taholah and Bethel parents approved most highly of their teachers 
while Neah Bay and Moclips parents were somewhat more critical. The Barnhardts 
did not agree, ranking the schools in order of parental approval of the teachers, 
from highest to lowest, as: Taholah, Neah Bay, Bethel, and Moclips. Connelly 

declined to rank the four schools on this dimension, although he did indicate 
that Taholah parents were somewhat more positive in their evaluations than the 
others . 
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The results on the scale concerned with parents* approval or disapproval 
of the school administration indicated that Taholah parents did not approve 
more highly of the Taholah administration than they did of the Moclips 
administration. Connelly felt that Taholah parents approved very highly of 
the Taholah administration but criticized the Moclips administration quite 
seriously. On the other hand, he did agree with the scale results that Neah 
Bay parents were quite critical of their administration. 

Finally, we concluded from the interview analysis that few students and... 
parents were very critical of the schools that served them, largely because 
of a lack of sophistication and knowledge in the area of education. We guessed 
that many parents felt that, if their children were getting passing grades in 
school and continuing to attend, the school must be !, 0 K n . The fact that the 
less sophisticated Bethel parents had the most consistently positive ratings 
seemed to support this. The field team, on the other hand, stressed a different 
point. Connelly noted that "much of the push for change is involved with a 
political drive for Indian autonomy and broader participation in decision- 
making." The Barrhardts also felt that native people most often provided the 
strongest push for meaningful change. There is certainly a difference between 
our interpretation of the scale results and the field team members* interpre- 
tations of their observations. Our different interpretations certainly reflect 
our respective biases as well as our different techniques of data-ga thering 
and analysis. 

We would like to reconcile rind synthes ize these two understandings of 
Indian communities if \ jssible. We feel that, on the one hand, few students 
ai.'d parents are voicing serious, well- thought criticisms of the school or the 
educational process; Indian communities are no different from most other American 
communities in this respect. On the other hand, some people--in Taholah and 
Neah Bay particular ly--evidently are raising serious criticisms. Moreover, the 
potential for more serious and meaningful criticism exists and may develop if 
the movement for more Indian control of Indian affairs takes hold in many Indian 
communities at the grass roots level. It also appears that we must not over- 
emphasize the teachers' dissatisfaction with certain aspects of the school, 
especially the curriculum. Teachers may be more aware of some of the faults 
of the schools in which they teach than most parents, but few seem prepared or 
able to do much about them. While many schools may make minor curricular 
changes (e.g. offering a course in local Indian history and culture), more 
fundamental changes leading to a seriously "bi-cultural" or Indian-oriented 
education are unlikely to be advocated by many teachers at the present time. 



Indian Attitudes toward Formal Education 



Students: Wc could not say that students in any one school were more 

interested in getting a formal education than students in any other school. 
The field workers did rank the students according to interest in school, but 
they did not agree with each other. ' 
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Parents; From the rating scale analysis we were unable to rank parents 
according to the degree of concern which they felt in each community for their 
children's education. The workers did rank them, but again they disagreed 
with each other as to which community was highest and which was lowest. 

Teachers; Based on four items on the Teacher Questionnaire, we concluded 
that the majority of teachers believed that Indian parents were concerned that 
their children learn in school, but that less than a third of the teachers in 
these schools felt that the students v/ere very interested in their education. 

The Barnhardts appear to agree with this. They indicated that, on the whole, 
parents were more concerned that their children get an education than the 
children were themselves. However, all field workers agreed that teachers in 
most communities --with the possible exception of Neah Bay--felt that Indian 
parents gave little actual support for continued achievement in school. Most 
also believed that Indian home life handicapped students as far as their success 
in school was concerned. We find this compatible with the interview and Question- 
naire analysis and are inclined to agree with the field team's judgment. 



Parents’ Knowledge of the School 

The scale results agree quite well w.-th the field team’s observations. 
According to the scale analysis, Taholali parents were most knowledgeable about 
the Taholah school; Neah Bay parents were next, while Bethel parents knew least 
about their schools. The field workers felt that Neah Bay and Taholah parents 
were equally knowledgeable (although in different areas) , while Bethel parents 
were least informed „ 



Teachers ; Understanding of the Community 

We based our conclusions on two teacher rating scales and found that 
teachers at Taholah, Neah Bay, and Bethel had a "good understanding of and a 
relatively broad sympathy for the Indian people and their problems," while 
teachers at Mo clips remained somewhat less understanding and sympathetic. On 
Scale A (Teachers) , the schools were ordered from most to least knowledgeable 
of the Indian community as; Neah Bay, Bethel, Taholah, and Moclips. On Teacher 
Scale B, the schools were ranked from most to least "sympathetic” and "under- 
standing" as follows; Bethel, Taholah, Neah Bay, and Moclips. Connelly did 
not feel that these schools could be ranked on these bases, noting instead that 
there were some well-informed and sympathetic teachers in all schools. The 
Barnhardts did rank the four schools and disagreed with our ranking based on 
the scale results. For both "knowledge" and "sympathy," they ranked teachers 
(from highest to lowest) as; Taholah, Neah Bay, Bethel, and Moclips, stressing 
that Taholah and Neah Bay were actually quite similar. This lack of agreement 
between the scales and the Barnhardts is disappointing, though we should note 
that the difference in mean scores between 'Taholah, Neah Bay, and Bethel teachers 
was less than *50 on both rating scales used for this dimension. Only Moclips 
teachers stood out as markedly different from teachers in the other schools. 

It is obvious that we should not draw conclusions from such small differences 
O in mean scores in the rating scale analysis. 
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Involvement: of the Community in School Affair s 

The field team’s observations agreed with the results on the parent 
scale dealing with involvement of the community in school affairs, but the 
field workers were able to provide more ' information about the specific 
situations in each community than the scales were. All agreed that Taholah 
parents were most involved with their school, Neah Bay parents next, and 
Bethel parents least. Comrell'y^noted that only in Taholah did the Indian 
community really control the school (through an all-Indian school board), 
though even there, the fact that few parents understood the school program 
and its goals made this control less meaningful. At the other extreme was 
Bethel, where the school was operated by the state of Alaska and where most 
native parents had very little comprehension of or involvement in school 
affairs , 



Conclusions 

With some significant exceptions, agreement between the rating scale and 
questionnaire analysis and the field workers’ observations on ranking and 
comparing the four schools was good. The main exceptions were: (1) In eval- 

uating parental opinions of the school administration in the four schools, 
the field team was in marked disagreement with the scale results; (2) In 
ranking the four schools according to the teachers’ “unders tanding of the 
local Indian community,” there was no agreement on the ranking of three of 
the four schools; (3) The scales sometimes failed to differentiate between 
schools where the field workers felt they could; and (4) The field workers 
sometimes felt that no meaningful differentiation between or ranking of schools 
could be done where we had done so with the scale results. 

The more important area of disagreement between the analysis of the inter- 
views and questionnaires and the field workers’ observations was in the inter- 
pretation of what each ”saw.” For example, the members of the field team felt 
that, though many teachers voiced support for a policy of “cultural pluralism” 
and wanted a course in Indian history and culture, few if any understood the 
native culture or the idea of n bi-cultural education.” Another major differ- 
ence in interpretations was in evaluating the degree of dissatisfaction in the 
native community (students and parents) , the school (teachers and administrators), 
and among influential persons in the community. We interpreted the scale results 
as indicating that students and parents voiced few serious criticisms of the 
schools while we saw teachers and community leaders as more critical and more 
apt to institute changes. The field team members felt that meaningful changes 
would only come as a result of agitation from the native community, and this 
they believed was possible. In addition to such major differences in interpre- 
tation, of course, the interpretation of the scale results was plagued by a 
lack of knowledge of the individual communities which would have "explained” 
the reasons for the differences in attitudes, opinions, and perceptions between 
different communities and between different members of the same community. 
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In answer to our original questions: Are the differences in the 

distributions of ratings and in mean "scores" between schools valid? 

Are these differences consistent with the observations of the field team?, 
the answer is a qualified yes* The qualification is that these differen- 
ces must be quite large to be considered valid. Even then, we must expect 
that a small percentage of our results may not be in accord with the field 
teams 1 observations. However, the question of how to interpret the results 
of this analysis--a vitally important but sticky problem in all such 
research as tliis--is difficult to evaluate* We recognize, of course, that 
both those who observed in the field and these who analyzed the interviews 
and interpreted the results of the ratings j.ncorporate their particular 
biases in their interpretations of their data. However, we do feel that 
skilled field observers have a better basis for interpreting much of the 
data than do those who simply read and rated the interviews. Our conclusion, 
then, is that field workers need to add their observations and interpretations 
to the results of the interview and questionnaire analysis in preparing the 
reports of this study. 
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